Example of a possible interpretation of Tsallis entropy 
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Abstract: We demonstrate and discuss the process of gaining information and show an example in which some specific way of 
gaining information about an object results in the Tsallis form of entropy rather than in the Shannon one. 
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Some time ago the notion of information in the form dictated by Shannon en- 
tropy [lj was used by us to study multiparticle production processes [2j. Later 
we found that its Tsallis version [3] is more suitable El , because the nonexten- 



sivity parameter q characterizing Tsallis entropy bears information about the 



intrinsic fluctuations in the physical system |B] I. The finding that q can be 



interpreted as a measure of such fluctuations resulted in a new subject called 
super 'statistics dealing with all kinds of fluctuation, see J7J. In all these studies 
we used either the Shannon or Tsallis form of information entropy without, 
however, any deeper understanding or argumentation about what in fact makes 
them so different, i.e., without deeper thoughts about the details of the process 
of gaining information leading to one or another form of information entropy. 

In this note we shall concentrate on this problem, using as our basic tool a 
widely known example presented in [H]) which demonstrates how to deduce in 
a simple manner the form of Shannon informational entropy by considering the 



* Actually, what we later call entropy S q = H, was introduced independently before, see for example 
[I] and then rediscovered again by Tsallis in thermodynamics. 



process of finding the location of some object in a prescribed phase space (like, 
for example, a point on a sheet of paper). We shall develop an equivalent pro- 
cedure resulting in Tsallis entropy instead. In particular we shall demonstrate, 
using these examples, how the way in which one collects information about an 
object decides the form of the corresponding information entropy. As already 
mentioned above we shall concentrate only on the comparison between Shannon 
P (5 = S q= \) and Tsallis [3j (S q = S q ^\) forms of entropy: 

(i - pi 1 ) 

S = ~J2Pi^Pl & S q = ~J2Pl^qPi = ~J2Pi ] _ l -■ (!) 

Acting in the same spirit as in [S] , consider a system of size Vo and divide it into 
cells of size V each; we then have M = Vq/V such cells (divisions). Suppose 
now that in one of these cells an object is hidden (we shall call it a particle in 
what follows) and that the probability to find it in a cell is the same for all cells 
and equal -h. The corresponding Shannon entropy, describing the situation of 
finding this particle in one of the cells, is: 

M 1 / 1 \ 

Suppose that the cells were formed by consecutively dividing previous cells into 
two equal parts and that we have performed / such divisions. Then M = 2* 
and Shannon entropy ([2]) corresponds conventionally to S = log 2 M = / bits 
of information. The tacit assumption is that to locate a particle i n th e system 



is equivalent to finding the respective cell containing this particl e**! . In such 
an approach, entropy equals just the number of YES/NO questions needed 
to locate the selected particle. In a sense, our particle is structureless, i.e., 
it has no additional features which would have to be investigated before its 
proper recognition; the localization of the particle is therefore equivalent with 
its recognition. 

Suppose, however, that the particles we are searching for have some additional 



Actually in [8] this particle was supposed to be pointlike and one attempted to find its location 
inside some system. To do this, this system was consecutively divided in halfs with a priori equal 
probabilities to find this particle (point) in one of the two cells and this procedure was continuing 
until the desired accuracy was obtained. In our case this accuracy dictates the number of cells into 
which our system is divided. 
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features one has to account for and that in a cell there can be more (or less) 
than one particle. It is obvious that in such a case the localization of the cell as 
performed above is not equivalent to the recognition of the right particle itself. 
One can now be faced with two situations: 

• One finds the cell with a particle in it but one is still not sure that this is the 
right ("true") particle; some additional search involving additional features 
mentioned above is required - one needs more information than in the usual 
case. 

• One recognizes the right particle already before the search of the proper cell 
is finished, it means that some information offered is redundant - one needs 
less information than in the usual case. 

The problem now is: how to quantify this problem? There is a priori an enor- 
mous number of factors which result in those additional features which should 
apparently be accounted for. On the other hand, from our point of view, all 
of them are, in a sense, identical because they simply transform the originally 
structureless particle to a particle endowed with some structure which can vary 
from one particle to the other. Let us therefore concentrate on a simplest pos- 
sibility and assume that it is enough to replace the single particle considered 
originally in [Sj by a number of identical particles endowed with some artificial 
size is, which can occupy each cell. Identical means therefore that all particles 
have the same size which they keep all the time. Notice that: 

• in the whole volume Vq considered one can only put N = Vq/v particles; 

• in a given cell one can only put k = V/v particles; it is very important to 
realize that one can have k > 1 as well as k < 1 (in addition to the original 
case corresponding to k = 1). 

As before we again attempt to locate the selected particle in our system in a 
most effective way (i.e., by using only the minimal possible amount of informa- 
tion). The probability to choose the cell with this particle is p = 1/M = V/Vq. 
However, now the size of the particle matters and the cell can be occupied by 
a number of particles among which we must choose the one we are looking for. 
As illustrated in Fig. Q] one can encounter three typical situations: 

• Even when one finds the right cell one still has to search for a while before 
deciding that the chosen particle is the right one. In our example this is 
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Fig. 1. Schematic illustration of three possible situation encountered when gathering information. Left 
panel: whatever number of divisions the object we are looking for is smaller then the actual cell and 
can require some additional (in comparison to the usual) search. Middle panel: there is no additional 
information needed to locate the object. Right panel: object shows itself (and therefore can also be 
identified) already before the usual amount of search (i.e., divisions) is performed. Only the middle 
panel situation leads to the Shannon form of entropy, the remaining two result in the Tsallis entropy 
instead. 



visualized by the fact that the particle is smaller then the cell and there can 
be more than one particle per cell (notice that more does not mean here that 
the actual number is an integer, it can be any positive number), cf. left panel 

of Fig. m 

• It can happen that one is sure that the chosen particle is the right one even 
before the right cell has been identified. In our example this corresponds to 
a situation when the particle is bigger than the cell. This means that the 
particle occupies more than one cell (again, more means that this is any real 
number greater than unity but smaller than the maximally allowed number 
of cells equal Vq/V), cf. right panel of Fig. Q3 

• The information needed to locate the particle is the same as to find the right 
cell. In our example it simply means that volumes of cell and particle are 
equal and there can be only one particle per cell., cf. the middle panel of Fig. 

m 

To continue along this line let us notice that one (chosen) particle can register 
in a cell with probability p or not register in this cell with probability 1 — p. 
In the case when this particle is not identified with the cell (as is the case in 
the case of counting the Shannon entropy), in this cell there can be maximally 
k — 1 other (false) particles. Assuming independence of events, the probability 
of the occurrence r such particles in the cell is p r . Therefore, on the average, 
the probability to register a false particle (and not the chosen one) per one false 
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particle ie. 



~k ~k ~k 



Notice that by doing so we are also tacitly assuming that any false choice 
also removes (or equivalently marks somehow) the falsely chosen particle as 
misidentification. 

Before proceeding further, a few words of conditions under which formula (J3j) 
may be valid are in order. Our picture could physically correspond to the sit- 
uation in which we perform measurements with noise (q > 1) or when errors 
connected with the measurement exceed the size of the cell (q < 1). In this case 
our object is identified in a number of cells (in other words, in this case our 
cells are not "mutually exclusive" as they were in the usual situation leading 
to Shannon entropy). One can also encounter a situation when the cells are not 
refined enough in phase space, equivalent to the case in which cells are exactly 
known but the location of our object (particle) is not fixed. All such situations 
eventually lead to Eq. (J3j). 

To continue, the question we have to answer now is: what is the corresponding 
entropy in this case, or - what is the analogy to YES/NO questions in the 
previous case, where the number of questions was the entropy? We argue that 
the analogy to YES/NO questions in this case is the sum over all cells of the 
probability to not register the particle (i.e., probability to register only false 
particles). Notice that Eq. ([2]) gives us the gain of information we get from a 
single cell. For a system of 2 cells, i = 1,2, with pi = and p2 = 1 one has 
H = 0, whereas for < p\ < 1 one gets H > 0. Therefore one can say that the 



* * * Here and below we are formally using the symbol of summation in spite of the fact that, as stated 
before, k = V/v is not necessary an integer. Therefore, when necessary, one has to use continuous k 
and replace summation by a suitable integration to calculate the corresponding (...) quantities, as 
for example, 



(Pfc-i) 



rz/i Ji p r dr for k > 1 
- ln(p) fl „r 



1-fc 



J k p r dr for k < 1 



P 



k-l 



P 



k-l 



-p ln(p). 



One gets the same limiting behavior directly applying the k — ► 1 limit to the integral. 
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Fig. 2. Illustration of the dependence of H on k = V/v for the number of cells equal to Vq/V = 2 5 
(full line). The Shannon entropy (dotted line) for the same number of cells is equal 5 = 5 (because 
we have now 5 bits). Notice that H > S for k < 1 and H < S for k > 1. 

entropy for a system of M cell is given by 

M M Jk-l _ i 

# = £<P*-i> = -£/t— T' ( 4 ) 

i=l i=l K — 1 

which in our case can be rewritten in the following form, 

H = -H^ KVo) y z • (5) 

Notice that entropy defined by Eq. (HJ) or (J5J) is formally identical in its form 
with the Tsallis entropy [3j with q = k = V/v. Its behavior is illustrated in Fig. 
[21 It has the following characteristic properties: 

• H — > if V — > Vo, i.e., when one has just one cell (in this limit there is no 
information left in the system, see Fig. [2], k > 1 case); 

• When v — ► Vo, i.e., when the particle we are looking for covers all cells 
(becoming therefore identical with the system itself), the entropy H grows 
monotonically and reaches the limiting value 

H = (M l - l l M - 1) < M. (6) 

M - 1 v ; w 

It is interesting to note that when M becomes very large then H — ► M 
in this limit. This should be contrasted with the much milder behavior of 
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Shannon entropy which in this limit becomes S oc InM (cf. Fig. [21 k < 1 
case). The difference arises because in this case in order to fully identify a 
particle in the system one must go through all cells with positive outcome, 
not only through a sunset of them as is done in the procedure leading to 
Shannon entropy. 

• H — > S = — E|£i In \yA if v — > V, i.e., when the particle we are looking for 
has the same size as the cell; in this case k — > 1 and we recover the classical 
definition of Shannon entropy as given by Eq. ([2]) (see intersection for k = 1 
in Fig. E]) . 

It is interesting to mention at this point that in the case where we would be 
interested in finding not one but some number, say k^-, of particles among k\ 
particles (notice that in the previous discussion k\ = k and k^ = 1) then Eq. 
(|3]) would be replaced by 

(-] _ n \h k x -k 2 i 

= i^k £ p " = irk/ 1 ~ " v 2 - x ) • (7) 

which would then result in the two parameter form of entropy i7, even more 
general than the Tsallis entropy (see [9] for examples and discussions of such 
entropies, we shall not pursue this point further here). 

Let us finally consider two systems consisting of M elements each: A and B. Let 
us proceed in the same way as above, treating each system independently with 
probabilities pa and pb replacing p. Suppose we are looking for two particles: 
one from A and one from B (and let us assume that they have the same structure 
in both systems). Notice that even when individually ]9 = pa - Pb, their average 
does not factorize but is given by the following expression: 

1 - PAPB r r VAVB /, fc _i k -l\ 

(pA,k-i ■ VB,k-i) = k _ 1 UPaVb = y (1 - Pa Pb ) = 

= PB{pA,k-l) +PA{pB,k-l) + (1 - k)(p Ai k-l){pB,k-l)- (8) 

This in turn means that (i denotes summation over cells in A and j in B) 

M M 

Ha,b = T,J2(PA,k-iPB,k-i) = H A + H B + (1 - k)H A H B , (9) 
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i.e., that entropy H is nonadditive. 



To summarize: we have demonstrated on a simple example how the way one gets 
information on the system leads to different forms of the information entropy 
when this is understood as some suitable measure of this information. The most 
general form, encompassing the situations in which the object we are looking for 
has some internal degrees of freedom (here summarily described by endowing 
it with some artificial size), is the one described by entropy H as defined by 
Eq. (j3J) which has the form of the Tsallis entropy [3]. The Shannon entropy 
([2]) is (at least in the example studied here) only a limiting case corresponding 
to a structureless object. As one can see in Fig. E] it corresponds to a single 
point only for which k = q = 1. Otherwise one always gets Tsallis entropy. One 
should bear in mind that this is a really very simple (if not simplistic) analysis, 
assuming only discrete situations. On the other hand, we argue that it already 
explains the essence of the difference between H = S q and S = S q= i in Eq. (CD) 
(and it also bears a potential for even further developments as witnessed by Eq. 

)• 



One must, however, bear in mind the possible limitation of our approach caused 
by our particular inclination towards problems of high energy multiparticle 
production processes where intrinsic fluctuations are very important in a proper 
description of systems in which some given initial amount of energy is converted 
into finally observed particles (hadrons) in the process called hadronization. 
This link is behind our, probably peculiar view of entropy and its connection 
with some physical processes. As far as we can tell, the classical works on 
entropies and their properties (like, for example, [4j, see also the most recent 
[TO] ) are rather mathematical in their form and scope, and, for example, 



review 



not directly applicable to the subject mentioned above. One should mention 
at this point previous attempts to extend Shannon entropy in which either 
nonadditivity of the entropy measure was important, not the additional one 
parameter [TT], or in which a two-parameter family of trace formula entropies 



were discussed C2 



The natural question coming to mind is about a possible application of the 
method proposed here to other types of entropy. Because of the enormous num- 
ber of possible entropies (see, for example, the list in [13] and in [101), this goes 



outside the limited goal of our work. Nevertheless, closing our presentation we 
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make a few comments concerning the widely used Renyi entropy. It also has 
an extra parameter (often denoted by q) , however, contrary to Tsallis entropy 
it is extensive. The meaning of the q parameter used in Renyi entropy is quite 
different from that in Tsallis entropy. By construction, as discussed in detail 
in |14| . Renyi entropy R q is sensitive to non-uniformity of the measure of the 
phase space with q being a kind of control parameter specifying the regions 
of phase space of interest. From the point of view of our procedure one could 
envisage the same procedure to get R q as for Shannon entropy (with YES/NO 
questions). Both are maximal at equipartition (pi = 1/M), and the maximum 
equals InM. The parameter q of this entropy starts to act when distribution 
under consideration is not uniform, otherwise R q is identical with Shannon en- 
tropy (for M cells one has S = R q = InM , whereas for Tsallis entropy it is 
S q = ln q M ). Tsallis and Renyi entropies are connected by (using the same q) 
S q = ln q [exp (R q )]. 



The final version of this work owes much to discussions at the Facets of Entropy 
workshop in Copenhagen (2007), which GW gratefully acknowledges. Partial 
support (GW) of the Ministry of Science and Higher Education under con- 
tracts 1P03B02230 and CERN/88/2006 is acknowledged. 
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